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Abstract 

The  purpose  of  this  research  was  to  examine  if  factors  related  to  questionnaire 
response  rate  in  the  general  public  also  affect  response  rate  in  military-only  samples. 
Very  little  research  pertaining  to  this  specific  group  was  located  during  the  literature 
review.  Four  response  rate  factors  taken  from  studies  conducted  in  the  general  public 
were  selected  for  this  research  topic:  questionnaire  length,  questionnaire  delivery  mode 
(postal,  e-mail,  web-based,  direct  administration,  and  mixed  mode),  use  of  advance 
notices,  and  use  of  follow-up  reminders.  Data  were  gathered  for  a  meta-analysis  of  73 
previously  published  studies  which  utilized  a  survey  or  questionnaire  to  collect  data  from 
a  military-only  sample.  After  the  data  were  analyzed,  only  two  response  rate  factors 
produced  significant  results  for  this  study:  survey  length  and  survey  delivery  mode. 
Results  were  discussed,  and  recommended  research  areas  and  conclusions  are  given. 
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A  META-ANALYSIS  OF  QUESTIONNAIRE  RESPONSE  RATES 


IN  MILITARY  SAMPLES 


I.  Introduction 


Background 

A  well-known  reality  of  our  current  society  is  that  consumers  are  frequently 
surveyed  by  government  agencies  and  commercial  entities  to  better  understand  consumer 
attitudes,  likes,  dislikes,  feelings,  etc.  However,  somewhat  lesser  understood  is  the  value 
these  organizations  place  on  survey  data;  for  example,  survey  data  are  so  important  to  the 
U.S.  Census  Bureau  that  the  agency  allocated  an  advertising  budget  of  $167  million  in 
order  to  promote  public  awareness  of  the  year  2000  decennial  census  (McCarthy  & 
Beckler,  2000).  The  Census  Bureau  requires  this  census  data  in  order  to  achieve  their 
strategic  goal,  despite  facing  budgetary  constraints  and  a  possible  overall  decline  in 
census  response  (U.S.  Census  Bureau,  2006).  Regardless  of  these  constraints,  the  survey 
process  and  the  data  are  important:  the  census  itself  is  mandated  by  the  U.S.  Constitution, 
and  census  data  are  utilized  by  the  federal  government  to  allocate  over  $200  billion  of 
federal  funds  each  year  (U.S.  Census  Bureau,  2006). 

On  a  more  personal  level,  our  daily  interaction  with  media  sources  such  as  the 
Internet  and  television  broadcasts  are  replete  with  survey  results  from  CNN  and  USA 
Today  surveys,  Gallup  polls,  and  Nielsen  ratings.  In  some  cases,  the  desired  data  are  so 
important  to  the  organization  that  top  leaders  make  personal  endorsements  in  an  attempt 
to  increase  participation  among  its  members.  For  instance,  General  Moseley,  the  Chief  of 
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Staff  of  the  Air  Force,  recently  requested  participation  for  the  Air  Force  Climate  Survey, 
specifically  asking  for  “direct,  honest,  and  candid  inputs  with  the  goal  of  making  our  Air 
Force  even  better”,  and  further  emphasizing,  “Your  frank  comments  about  our  common 
Air  Force  culture  are  key  to  our  future  success”  (2006).  While  such  high-visibility 
surveys  are  more  notable,  there  are  likely  vast  quantities  of  surveys  administered  at 
various  lower  levels  within  organizations.  For  instance,  at  the  squadron  level, 
commanders  may  request  a  local  climate  assessment  survey  to  be  issued  to  squadron 
members  to  gauge  issues  pertaining  to  their  respective  squadrons  (Figlock,  2004).  Post¬ 
graduate  students  and  other  researchers  may  use  surveys  in  order  to  gather  pertinent 
information  for  their  respective  research  topics. 

Given  these  familiar  examples,  which  are  not  exhaustive  by  any  means,  one  can 
easily  appreciate  a  cost  which  might  be  associated  with  obtaining  such  data:  although 
useful  and  familiar,  surveys  incur  some  type  of  cost  to  the  developer  and  user  of  the 
survey,  and  developers  must  keep  within  their  limited  resources  (Groves,  1990).  In  fact, 
organizations  spend  considerable  resources  to  develop  and  distribute  surveys  in  an 
attempt  to  maximize  survey  responses  to  ensure  representative  samples  are  collected 
(Groves,  1990).  It  is  reasonable  to  assume  that  organizational  leaders  expect  some  return 
on  this  investment  into  the  survey  process  because  this  data  may  be  used  to  make 
decisions  which  could  positively  or  negatively  impact  the  performance  of  the 
organization.  For  example,  if  a  survey  generates  a  low  response  rate  or  non-response 
rate,  the  results  may  render  a  non-statistically  representative  sample,  thus  wasting 
organizational  resources  (Groves,  1990;  Porter,  2004).  Therefore,  one  goal  of 
organizations  which  utilize  a  survey  tool  is  to  maximize  survey  response  rate  in  order  to 
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increase  their  return  on  resources  invested  into  the  survey  process,  with  the  ultimate  goal 
of  improving  organizational  performance  (Karrasch,  2003). 

Surveys  have  been  extensively  studied  (Dillman,  1978;  Steeh,  1981)  and  as  a 
result,  researchers  have  learned  much  about  how  survey  characteristics  can  influence 
response  rates  (Dillman,  2000;  Cycyota  &  Harrison,  2006;  Groves,  1990;  Porter,  2004; 
Roth  &  BeVier,  1998;  Simsek  &  Veiga,  2001).  However,  despite  an  increasing  level  of 
knowledge  of  the  factors  affecting  survey  response  rates,  researchers  continue  to  struggle 
with  decreasing  response  rates  as  indicated  by  a  decline  during  a  10-year  period  from 
1990-1999  (Atrostic,  Bates,  Burt,  Silberstein  &  Winters,  1999;  Baruch,  1999;  de  Heer, 
1999);  further,  response  rates  may  have  been  on  a  decline  from  as  far  back  as  1952 
(Steeh,  1981).  In  fact,  some  companies  which  rely  on  customer  feedback  have  been 
forced  to  offer  monetary  incentives  to  increase  response  rate;  in  2002,  Singer  and  Kulka 
(citing  James  &  Bolstein,  1992)  reported  a  $1  incentive  to  be  the  most  cost  effective,  but 
reported  values  increased  to  a  few  dollars  (Goritz,  2004),  all  the  way  to  a  drastic  example 
for  highly  desirable  data  (e.g.,  $5),  especially  within  certain  population  demographics, 
such  as  young  adults  and  small  ethnic  groups  (Charlebois,  2006). 

Other  factors  complicate  the  issue  of  survey  response  rate.  First,  most  survey 
studies  sampled  almost  entirely  from  only  a  few  general  population  types: 
students/academia,  the  consumer  marketplace,  and  civilian  organizations  (Baruch,  1999; 
Simsek  &  Veiga,  2001;  Porter,  2004).  Second,  although  a  few  studies  (Cole,  2005; 
MacElroy,  Milucki,  &  McDowell,  2002;  Franke,  2001)  researched  the  effect  of  delivery 
mode  (e.g.,  postal  mail,  e-mail,  web-based)  on  survey  response  rate,  contradictory  results 
were  frequently  presented;  typically,  the  focus  was  on  one  delivery  mode  against  another 
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delivery  mode  instead  of  examining  the  overall  effect  of  multiple  modes  to  determine  the 
optimal  delivery  mode  (or  modes)  to  improve  survey  response  rate.  Last,  military 
organizations  were  rarely  identified  as  a  potential  population  for  significant  survey 
analysis  (Roth  &  BeVier,  1998;  Asiu,  Antons  &  Fultz,  1998).  Given  the  overall  trend  of 
declining  response  rates  and  the  almost  non-existent  research  on  response  rates  related  to 
military-only  populations,  this  topic  appears  to  be  an  area  which  warrants  further  study. 

Problem  Statement 

The  literature  review  revealed  a  general  lack  of  research  pertaining  to  survey 
response  rate  in  military-only  populations.  This  void  of  research  may  be  significant. 

First,  some  of  the  factors  affecting  survey  response  rate  may  not  apply  to  a  military-only 
population.  For  example,  while  monetary  incentives  (such  as  cash  or  redeemable  bonus 
points)  have  been  utilized  in  the  consumer  marketplace,  this  type  of  incentive  is  not 
typically  seen  in  a  military-only  survey  (Goritz,  2004).  Second,  during  these  times  of 
decreasing  Department  of  Defense  budgets  and  implementation  of  “Force  Shaping” 
reduction  in  force  measures  by  the  military,  the  requirement  exists  to  retrieve  as  much 
information  as  possible  from  a  shrinking  military  personnel  pool.  Last,  some  branches  of 
the  military  and  programs  within  the  military  require  use  of  surveys.  For  example,  Army 
Regulation  600-20,  dated  07  June  2006,  requires  a  unit  climate  assessment  be  completed 
within  90  days  following  a  change  of  command.  If  conducting  a  survey  is  a  regulation- 
driven  requirement,  and  the  goal  is  to  receive  quality  and  quantity  of  responses,  perhaps 
further  investigation  into  the  survey  process  for  military  populations  might  be  beneficial. 
Also,  military  students  participating  in  post-graduate  degree  programs  or  professional 
military  education  programs  frequently  utilize  survey  data  collection  as  part  of  their 
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education  and  research  process.  For  these  reasons,  the  factors  affecting  survey  response 
rate  on  military-only  population  will  be  the  focus  of  this  thesis  research. 

After  reviewing  the  literature,  many  factors  were  found  to  increase  survey 
response  rate  in  general  populations.  The  hypothesis  statements  will  postulate  how 
certain  response  rate  factors  may  potentially  influence  military-only  populations.  Factors 
such  as  survey  length,  mode  of  survey  delivery,  and  utilization  of  advance  notices  and 
follow-up  reminders  (all  previously  identified  in  the  literature  as  factors  affecting 
response  rate)  will  be  tested. 

Given  that  the  military  seems  to  be  an  underutilized  survey  population  for  survey 
response  rates,  by  using  published  research  studies  of  survey  response  rate  factors,  a 
meta-analysis  will  be  performed  by  examining  factors  influencing  survey  response  rate 
with  respect  to  a  military  population.  Electronic  databases  will  be  searched  for  any 
published  research  articles  which  utilized  a  survey  or  questionnaire  to  gather  infonnation 
in  a  military-only  population.  A  matrix  will  be  developed  to  show  the  previously 
identified  survey  response  factors  as  compared  against  each  individual  study,  annotating 
the  existence  of  each  unique  factor  (or  factors)  referenced  in  each  specific  published 
research  (each  treated  as  one  “case.”)  The  data  will  then  be  evaluated  via  regression 
analysis  in  order  to  show  the  relative  importance  of  the  factors  selected  for  this  research. 
Once  the  results  have  been  analyzed,  the  goal  of  this  research  is  to  highlight  certain 
response  rate  factors  which  may  improve  response  rates  for  military-only  populations.  At 
the  local  level,  this  infonnation  may  be  useful  to  students  at  the  Air  Force  Institute  of 
Technology  when  generating  surveys  and  questionnaires  as  part  of  their  data  gathering 
process  for  their  respective  research  areas.  At  a  higher  level,  the  results  might  help 
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development  of  squadron  climate  assessment  surveys  with  the  goal  of  increasing 
response  rates.  In  the  end,  the  overall  goal  is  to  help  identify  which  response  rate  factors 
produce  the  highest  response  rates  for  any  studies  which  seek  to  acquire  information  from 
a  military-only  population. 
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II.  Literature  Review 


Survey  response  rate  data  has  been  researched  as  far  back  as  the  1950s  (Steeh, 
1981)  and  continues  today  (Figlock,  2004;  Porter,  2004;  Cycyota  &  Harrison,  2006; 
Cycyota  &  Harrison,  2002).  The  reason  is  clear:  researchers  who  utilize  surveys  as  a 
primary  data  collection  means  are  concerned  about  maximizing  the  amount  of  data 
returned  to  them  in  the  form  of  surveys;  they  must  also  ensure  that  the  responses  obtained 
from  the  surveys  are  representative  of  the  population  from  which  it  was  drawn  (Groves, 
1990;  Porter,  2004).  However,  maximizing  the  return  rate  is,  in  practice,  a  formidable 
task.  Several  studies  show  that  survey  response  rates  were  on  the  decline  during  a  period 
from  1990  to  1999  (Atrostic,  Bates,  Burt,  Silberstein  &  Winters,  1999;  Baruch,  1999;  de 
Heer,  1999)  and  may  have  been  on  the  decline  since  the  early  1950s  (Steeh,  1981). 

According  to  Groves,  Fowler,  Couper,  Lepkowski,  Singer,  and  Tourangeau 
(2004),  the  use  of  a  survey  or  questionnaire  as  a  data  gathering  tool  most  likely  dates 
back  to  the  late  1800s.  For  example,  Charles  Booth,  a  wealthy  businessman  and  self- 
taught  sociologist,  surveyed  the  poor  and  indigent  of  London  during  the  1890s;  he 
ultimately  produced  a  17-volume  set  of  data  from  these  surveys,  titled  “Life  and  Labour 
of  the  People  of  London”  (Osborne  &  Rose,  2004).  However,  while  surveys  may  have 
appeared  in  social  research  in  the  1800s,  Groves,  et  al.  (2004)  also  states  the  development 
of  survey  methodology  did  not  begin  to  appear  until  the  1930s  and  1940s  in  large 
government  organizations,  being  further  refined  by  departments  such  as  the  U.S.  Bureau 
of  the  Census.  The  war  effort  during  the  early  1940s  (and  the  post-war  years)  contributed 
greatly  to  the  design  and  use  of  survey  methodology. 
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However,  shortly  after  this  structured  approach  began  to  define  survey 
methodology,  an  interesting  phenomenon  began  to  occur:  starting  in  the  1950s  and 
continuing  through  the  late  1970s,  a  decline  in  survey  response  rates  began  to  emerge, 
according  to  a  study  by  Steeh  (1981).  Similar  studies  evaluating  the  time  period  from 
1990  through  1999  reached  the  same  conclusion  (Atrostic,  Bates,  Burt,  Silberstein  & 
Winters,  1999;  Baruch,  1999;  de  Heer,  1999).  These  findings  of  declining  survey 
response  rates  prompted  researchers  to  begin  studying  possible  relationships  between 
survey  design  and  effects  on  response  rates;  many  were  examined  by  utilizing  a  meta¬ 
analysis  of  survey  design  research  (Groves,  1990;  Porter,  2004;  Roth  &  BeVier,  1998; 
Simsek  &  Veiga,  2001;  Cycyota  &  Harrison,  2006;  Cycyota  &  Harrison,  2002;  Fox, 
Crask  &  Kim,  1988). 

Overall,  these  studies  cited  several  factors  which  appear  to  impact  survey 
response  rates;  most  are  based  on  Dillman’s  survey  design  research  published  in  1978. 
Dillman’s  concept  espoused  the  “total  design”  concept  (updated  to  “tailored  design” 
approach  or  “tailored  design  method”  by  Dillman  in  2000)  specifically  for  postal  mail 
surveys  (Cycyota  &  Harrison,  2002;  Kaplowitz,  Hadlock  &  Levine,  2004).  Examples  of 
these  factors  listed  by  Dillman  and  others  include  survey  design  (length,  time  to 
complete,  and  ease  of  use),  survey  delivery  mode  (postal  mail,  e-mail,  web/Intemet 
based,  or  a  combination  of  modes),  issue  salience,  advance  notice  and  follow-up 
reminders  (number  of  contacts),  anonymity  and  confidentiality,  availability  of  final 
results  to  participants,  monetary  incentives,  and  sponsorship  level  of  the  survey  (Dillman, 
1978;  Roth  &  BeVier,  1998;  Cycyota  &  Harrison,  2006;  Cycyota  &  Harrison,  2002;  Fox 
et  al,  1988).  These  factors  include  a  wide  variety  of  variable  types  (nominal,  ordinal, 
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interval  and  ratio)  which  capture  different  types  of  data.  Some  variables  are  objective 
and  quantitative,  such  as  length,  but  others  are  more  subjective,  such  as  issue  salience. 

The  following  is  a  brief  discussion  of  some  of  the  many  factors  that  influence  survey 
response  rates. 

Issue  Salience 

Surveys  and  studies  which  appear  to  offer  a  respondent  a  measure  of  perceived 
value  or  an  opportunity  for  improvement  (i.e.,  salience)  appear  to  increase  survey 
response  rate  (Groves,  Cialdini  and  Couper,  1992).  Issue  salience  is  a  factor  which  may 
initially  appear  easy  to  measure:  how  salient  was  the  specific  content  or  topics  included 
in  the  survey  to  each  respondent?  While  issue  salience  has  the  potential  to  cause  the  most 
damage  to  survey  data  collection  via  non-response  (Wolford,  1994),  salience  of  an  issue 
is  subjective  and  difficult  to  measure  (Sheehan,  2001;  Cycyota  &  Harrison,  2002).  One 
way  researchers  have  tried  to  measure  issue  salience  is  by  using  researchers’  ratings  of 
respondents’  salience  using  an  ordinal,  three-tier  scale:  “1”  for  salience  to  a  general 
population,  “2”  for  salience  to  a  sub-population,  and  “3”  for  salience  to  a  group  within 
the  sub-population  (Porter,  2004;  McCarty,  House,  Harman  &  Richards,  2006). 

However,  even  these  researchers  have  admitted  that  their  process  of  assigning  a  value  to 
issue  salience  is  subjective  because  the  final  ranking  is  solely  detennined  by  the 
researchers,  not  the  respondents.  In  addition,  survey  designers  may  find  the  issue 
salience  factor  is  out  of  their  control,  and  yet  it  still  may  contribute  to  survey  non¬ 
response  rate  (McCarty  et  al,  2006).  Early  analysis  of  the  salience  factor  indicates  the 
need  for  additional  research  in  this  area  (Roth  &  BeVier,  1998). 


9 


Anonymity  and  Confidentiality 

Anonymity  and  confidentiality  are  two  separate,  but  related,  factors  which  impact 
survey  response  rate,  and  the  complexities  of  each  are  difficult  to  measure;  thus  far, 
different  studies  arrive  at  opposing  viewpoints  as  to  the  actual  impact  of  each  on  response 
rate  (Porter,  2004;  McCarthy  &  Beckler,  2000). 

Simsek  and  Veiga  (2001)  found  that  anonymity  and  confidentiality  are  highly 
complex  issues  comprised  of  social  desirability,  social  settings,  and  perceived 
consequences  caused  by  legally-retrieved  survey  responses  contained  in  e-mail 
transmissions  and  web-based  surveys.  Additionally,  a  worker’s  level  of  desired 
anonymity  did  not  merely  apply  to  an  individual  survey,  but  is  actually  a  personality 
characteristic  that  is  directly  related  to  the  individual’s  competence  level,  method  of 
interaction  with  people,  and  job  enthusiasm  level  (Simsek  and  Veiga,  2001).  Perceptions 
of  anonymity  and  confidentiality  may  also  be  affected  by  method  of  delivery  (Roth  & 
BeVier,  1998).  For  example,  when  surveys  are  handed  out  by  supervisors,  some 
respondents  may  feel  an  obligation  to  participate  in  the  survey  due  to  the  proximity  of 
their  supervisors,  but  then  provide  non-truthful  results.  In  addition,  mailed  surveys  may 
leave  respondents  with  perceptions  of  being  tracked  for  survey  completion,  even  when 
given  an  “anonymous”  user  identification  code  designed  to  promote  anonymity; 
completed  surveys  sent  back  to  a  specific  e-mail  account  may  incur  the  same  perceptions 
(Simsek  and  Veiga,  2001).  Last,  perceived  confidentiality  of  sensitive  data  may  impact 
respondents’  willingness  to  participate  (McCarthy,  Johnson  &  Ott,  1999).  All  the 
findings  by  these  researchers  indicate  a  wide  range  of  response  to 
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anonymity/confidentiality  issues  with  respect  to  survey  research  and  therefore  indicate 
further  research  in  this  area  should  be  explored  by  trained  psychology  professionals. 

Availability  of  Final  Results 

Sheehan  (2001)  noted  in  many  electronic  surveys,  the  promise  of  a  study’s  results 
in  a  future  publication  was  announced  to  the  survey  respondent  as  possible  motivation  to 
take  the  survey.  Goritz  (2004)  also  tested  respondents  for  interest  in  the  final  results  of 
the  survey.  While  survey  participants  have  expressed  interest  in  how  their  data  from  a 
survey  had  been  used,  many  of  these  requests  were  received  after  the  survey  had  been 
completed  (McCarthy,  Johnson  &  Ott,  1999).  Simsek  and  Veiga  (2001)  briefly  mention 
“an  offer  of  survey  results”  as  a  factor,  but  fail  to  mention  it  as  a  significant  factor.  In 
these  instances,  the  availability  of  final  survey  results  was  included  as  part  of  several 
other  factors  under  consideration,  but  was  never  again  identified  as  a  major  contributor  to 
survey  response  rates  (Simsek  &  Veiga,  2001). 

Monetary  Incentives/Compensation 

Singer  (2002)  found  monetary  incentives  increased  response  rate  for  participants 
perceiving  little  issue  salience,  but  contributed  almost  nothing  toward  increasing  response 
rate  for  participants  expressing  high  issue  salience.  Goritz  (2004)  ran  experiments  using 
a  monetary  raffle  as  an  incentive  to  complete  a  survey  against  surveys  without  a  raffle 
and  found  that  the  raffle  did  not  significantly  influence  responses.  Sheehan  (2001) 
observed  that  while  some  mail  surveys  in  the  past  have  contained  a  small  monetary 
incentive  for  participation,  with  the  advent  of  e-mail  and  web-based  surveys,  this  “pre¬ 
paid”  option  available  to  mail  surveys  was  now  becoming  difficult  to  implement  for  the 
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e-mail  and  web-based  surveys.  Porter  (2004)  found  similar  findings:  surveys  with  a  pre¬ 
paid  incentive  were  more  effective  than  post-paid  incentives  (i.e.,  incentives  received 
after  the  survey  was  completed.)  Different  avenues  are  being  explored  as  to  how  this 
incentive  process  can  become  more  effective  for  e-mail  and  web-based  surveys,  but  very 
little  research  has  compared  the  effects  (and  implementation)  of  monetary  incentives 
across  all  survey  delivery  modes,  and  thus  little  data  are  available  to  make  substantial 
comparisons  (Goritz,  2004). 

Survey  Sponsorship  Level 

Another  factor  affecting  survey  response  rate  is  the  level  at  which  the  survey  or 
study  receives  sponsorship  or  endorsement.  As  with  many  of  the  previously  mentioned 
survey  response  rate  factors,  a  review  of  the  literature  revealed  findings  which  support 
both  (or  neither)  theories.  For  example,  one  study  found  that  while  researchers  might 
assume  sponsorship  by  the  government  or  a  university  increases  response  rate,  this  study 
found  little  evidence  to  support  this  assumption  (McCarthy,  Johnson  &  Ott,  1999). 
However,  in  Porter’s  study  (2004),  he  did  find  that,  in  general,  respondents  are  more 
likely  to  respond  to  a  survey  sponsored  by  the  government  or  a  university  than  a  survey 
sponsored  by  the  commercial  sector.  As  previously  mentioned,  though,  as  has  been  the 
case  with  several  of  the  other  survey  response  factors,  the  overall  data  pertaining  to  the 
effect  of  sponsorship  level  on  response  rate  is  inconclusive.  For  example,  Roth  and 
BeVier  (1998)  described  two  studies  supporting  the  idea  that  sponsorship  contributed  to 
survey  response  rate  (Bruvold,  Comer  &  Rospert,  1990;  Fox,  Crask  &  Kim,  1988),  but 
also  detailed  another  study  (Yammarino,  Skinner  &  Childers,  1991)  which  found  either 
little  or  no  contribution  from  sponsorship  toward  increasing  survey  response  rate. 
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The  literature  has  indicated  a  need  for  further  research  of  each  of  these  previously 
discussed  factors.  However,  while  an  investigation  of  these  factors  would  no  doubt  be  of 
value,  such  variables  are  more  difficult  to  obtain  and  are  therefore  less  readily  available 
in  the  literature.  Considering  that  this  study  will  rely  on  meta-analytic  data,  it  is 
necessary  to  constrain  the  scope  of  variables  to  those  that  are  readily  available  and 
quantifiable  from  the  literature.  The  following  section  describes  the  factors  which  will  be 
considered  in  this  study:  survey  length,  delivery  mode,  advance  notice,  and  follow-up 
reminders. 

Survey  Length 

One  frequently  investigated  factor  affecting  survey  response  rate  is  survey  length 
(Dillman,  Sinclair  &  Clark,  1993;  Bogen,  1996).  Intuitively,  the  impact  of  survey  length 
on  response  rate  may  seem  obvious:  the  longer  the  survey,  the  lower  the  response  rate 
(Sheehan,  2001).  Smith,  Olah,  Hansen,  and  Cumbo  (2003)  found  nearly  a  doubling  of 
response  rate  during  their  study  comparing  a  one  page  survey  against  a  similar  three  page 
survey  covering  the  same  topics.  Similarly,  Dillman  et  al  (1993)  found  that  survey  length 
negatively  impacted  survey  response  rate.  The  result  is  basically  a  negative  linear 
relationship,  with  one  major  exception:  when  a  survey’s  length  was  reduced  to  five 
questions  or  less,  the  response  rate  began  to  decrease  (Dillman  et  al,  1993).  This  implies 
the  linear  relationship  no  longer  exists  for  detennining  response  rate  based  on  survey 
length  when  the  survey  is  composed  of  five  or  fewer  questions.  However,  for  this  study, 
all  the  surveys  and  questionnaires  selected  for  final  analysis  will  contain  more  than  five 
questions  which  will  allow  analysis  to  be  performed  in  the  region  which  is  basically 
linear. 
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Additional  studies  offer  more  support  for  the  impact  of  survey  length  on  response 
rate.  For  example,  potential  respondents  are  interested  in  the  amount  of  time  a  survey 
will  take  to  complete  (Bogen,  1996;  Groves,  Cialdini  &  Couper,  1992).  Yu  and  Cooper 
(1983)  and  Heberlein  and  Baumgartner  (1978)  conducted  quantitative  studies  in  these 
areas,  and  their  results  reached  the  same  conclusion:  longer  surveys  generally  produce 
lower  response  rates. 

Given  these  studies  on  how  survey  length  affects  response  rate,  the  initial  research 
question  will  bring  forth  the  concept  that  military  samples  will  act  the  same  as  the  general 
population  samples  surveyed  from  the  research  previously  discussed.  While  these  studies 
were  not  directed  at  military-only  samples  (and  there  appears  to  be  almost  no  research 
specific  to  how  survey  length  affects  response  rate  in  military  samples),  the  experts  in  the 
field  of  survey  response  rates  appear  to  agree  that  in  most  cases,  longer  surveys  will 
decrease  survey  response  rates. 

Based  on  the  studies  pertaining  to  survey  length  as  a  factor  affecting  survey 
response  rates,  the  following  hypothesis  statement  is  proposed: 

Ha,:  Survey  length  is  related  to  response  rates  in  military  samples,  while 
controlling  for  all  other  variables. 

Conversely,  two  major  studies  reached  conflicting  conclusions  as  to  the  actual 
effect  of  survey  length  on  response  rate  (Bogen,  1996;  Sheehan,  2001).  Bogen  (1996) 
referenced  a  small  number  of  studies  which  either  found  no  impact  or  minimal  impact 
from  survey  length  on  response  rate;  however,  several  of  these  studies  examined  surveys 
in  which  both  a  long  version  and  a  short  version  were  completed  in  less  than  5  minutes. 
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These  types  of  surveys  would  probably  best  be  described  as  “short”  based  on  the  small 
amount  of  time  required  to  compete  either.  Sheehan  (2001)  referenced  a  study  by  Bean 
and  Roszkowski  (1995)  in  which  longer  surveys  actually  produced  a  somewhat  elevated 
response  rate  as  compared  to  the  shorter  surveys. 

Survey  length  may  not  have  a  substantial  impact  on  military  members  because 
they  may  be  perceived  as  having  greater  organizational  commitment.  Gade  (2003) 
mentions  a  military  member’s  support  of  the  military  as  not  just  a  job,  but  actually  a 
“calling”  that  far  exceeds  a  regular  job.  This  level  of  organizational  commitment  could 
lead  a  military  member  to  feel  more  obligated  to  participate  in  and  complete  surveys, 
regardless  of  the  length  of  the  survey.  This  area  for  exploration  is  stated  in  the  following 
hypothesis  statement: 

Hoji  Survey  length  is  not  related  to  response  rates  in  military  samples,  while 
controlling  for  all  other  variables. 

Survey  Delivery  Mode 

Several  types  of  survey  delivery  modes  will  be  considered  in  this  study:  Internet 
(or  web-based),  e-mail,  paper  and  pencil  surveys  (composed  of  postal  and  direct 
administered),  and  various  combinations  thereof  (multi-mode.)  While  paper  surveys  are 
known  to  have  been  in  use  since  the  early  1900s,  the  infusion  of  the  Internet  into 
American  culture  since  the  mid-1990s  has  created  opportunity  for  electronic  surveys, 
both  web-based  surveys  and  e-mail  surveys  (Weible  &  Wallace,  1998;  Zhang,  1999).  An 
analysis  in  2002  found  almost  60%  of  Americans  had  access  to  the  Internet;  accordingly, 
the  rise  in  the  use  of  electronic  surveys  (either  e-mail  or  Internet-based)  rose  as  well 
(Lenhart,  Horrigan,  Rainie,  Allen,  Boyce,  Madden  &  O’ Grady,  2003;  MacElroy,  Milucki 
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&  McDowell,  2002).  Surprisingly,  different  survey  delivery  modes  may  offer  different 
levels  of  data.  MacElroy,  Milucki,  and  McDowell  (2002)  found  that  in  a  comparison 
between  a  web-based  survey  and  a  pencil  and  paper  survey,  when  respondents  were  given 
the  option  to  choose,  the  results  from  open-ended  responses  obtained  from  the  web-based 
survey  contained  more  detail  and  were  written  at  a  higher  education  level.  Also, 
electronic  surveys  may  offer  the  ability  to  collect  more  data  than  paper  surveys.  For 
example,  the  use  of  hyper-text  markup  language  (HTML)  in  a  web-based  survey  enables 
variable  color,  font,  and  graphics  enhancements  which  are  typically  unavailable  or  not 
cost  effective  in  paper  surveys;  survey  designers  may  also  use  HTML  code  to  prevent 
respondents  from  entering  incomplete  or  duplicate  data,  as  well  as  requiring  responses  to 
certain  questions  before  continuing  with  the  survey  (Schleyer  &  Forrest,  2000). 

Access  to  computer  resources  for  military  populations  (both  at  work  and  at  home) 
is  currently  at  an  all-time  high.  One  indication  of  the  ever-increasing  computer  resource 
demand  was  the  implementation  of  the  Common  Access  Card  (CAC),  directed  by  former 
Deputy  Secretary  of  Defense  (DEPSECDEF),  Dr.  John  Hamre,  in  a  1999  memo  titled 
“Smart  Card  Adoption  and  Implementation”  (GAO,  2003).  These  cards  will  be  used  by 
over  4  million  military  and  Department  of  Defense  personnel  and  will  primarily  be 
utilized  to  control  access  to  computing  devices  and  networks  (desktop,  laptops,  and  other 
networked  devices)  and  also  regulate  access  to  controlled  areas  (GAO,  2003).  Using  the 
large  number  of  CAC  cards  as  an  indicator  of  the  number  of  computers  the  military  uses 
on  a  day  to  day  basis  implies  that  most  military  members  readily  have  access  to 
computers  in  their  daily  work  activities.  Therefore,  the  access  availability  for  military 
populations  and  the  rise  in  utilization  of  the  electronic  survey  (MacElroy,  Milucki  & 
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McDowell,  2002)  leads  to  the  following  hypothesis  statement  relating  delivery  modes  to 
survey  response  rate: 

Ha2:  Delivery  mode  does  influence  response  rates  in  military  samples,  while 
controlling  for  all  other  variables. 

Conversely,  by  focusing  on  one  type  of  delivery  mode  without  fully 
understanding  its  impact  on  the  surveyed  population,  a  researcher  may  miss  or  lose  data 
from  a  segment  within  that  population.  One  concern  regarding  different  delivery  modes 
is  the  availability  of  computer  resources  for  a  certain  portion  of  a  population.  The  term 
“digital  divide”  describes  the  difference  between  respondents  who  own  or  have  access  to 
computer  resources  required  for  an  electronic  survey  as  compared  to  those  who  do  not 
own  or  do  not  have  computer  access  (Lenhart,  et  al.,  2003).  This  disparity  can  be 
attributed  to  differences  between  age,  income,  and  geographic  locations  (Schleyer  & 
Forrest,  2000). 

Pencil  and  paper  surveys  appear  to  hold  interest  in  certain  populations.  Older 
generations  are  less  interested  in  participating  in  electronic  surveys,  either  e-mail  or  web- 
based,  than  in  a  traditional  paper  survey  (Lenhart,  et  al.,  2003;  Kaplowitz,  Hadlock  & 
Levine,  2004).  Also,  Mehta  and  Sivadas  (1995)  found  a  slightly  higher  response  rate  for 
mailed  surveys  rather  than  electronic  surveys  for  one  military  sample  (Adams,  1996). 
Two  studies  found  different  patterns  of  responses  when  subjects  responded  to  a  paper 
survey  versus  an  electronic  survey  (McCoy,  Marks,  Carr  &  Mbarika,  2004;  Webster  & 
Compeau,  1996).  However,  a  study  conducted  by  Franke  in  2001  found  almost  no 
difference  between  similar  paper  and  electronic  surveys.  In  another  example,  the 
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Defense  Manpower  Data  Center  (DMDC)  conducts  nine  web-only  “Status  of  Forces” 
surveys  each  year  (DMDC,  2006)  and  also  many  surveys  given  by  AFIT  thesis  students 
(e.g.,  Martinson,  2005)  utilized  only  web-based  surveys,  given  their  populations  of 
interest  all  have  official  e-mail  accounts  and  access  to  computers  at  work.  However,  by 
utilizing  only  one  survey  delivery  mode,  researchers  assume  the  risk  of  survey  non¬ 
response  for  any  segments  within  populations  who  don’t  prefer  that  specific  delivery 
mode.  Therefore,  if  researchers  are  interested  in  capturing  data  from  broad,  general 
populations,  perhaps  merely  using  only  one  survey  delivery  mode  is  insufficient,  and 
multiple  delivery  modes  (or  mixed-modes)  should  be  considered  in  order  to  collect  as 
much  data  as  possible  (Kaplowitz,  Hadlock  &  Levine,  2004;  Dillman,  2000;  Schaefer  and 
Dillman,  1998).  This  leads  to  the  following  hypothesis  statement: 

Ho2:  Delivery  mode  does  not  influence  response  rates  in  military  samples,  while 
controlling  for  all  other  variables. 

Advance  Notice  and  Follow-Up  Reminders 

Advance  notice  and  follow-up  reminders  (pre/post-contact)  have  shown  to  almost 
always  produce  positive  results  toward  increasing  response  rates  (Groves,  et  ah,  2004; 
Fox,  et  ah,  1988;  Yammarino,  et  ah,  1991).  Roth  and  BeVier  (1998)  surmised  that 
advance  notice  and  follow-up  reminders  are  almost  synonymous  and  that  advance  notice 
could  be  counted  as  one  follow-up  notice  instead  of  being  counted  as  an  advance  notice. 
Some  of  the  research  has  gone  as  far  as  recommending  a  simple  fonnula  to  determine  the 
number  of  reminders  and  length  of  time  between  reminders  (Roth  &  BeVier,  1998; 
Martin,  Duncan,  Powers  &  Sawyer,  1989).  Also,  in  one  very  specific  example, 
Kaplowitz,  Hadlock  &  Levine  (2004)  found  that  an  advance  notice  (or  prenotice,  as  they 
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prefer  to  call  the  factor)  sent  by  postal  mail  increased  the  response  rate  for  a  web-based 
survey.  Additionally,  two  additional  studies  found  that  advance  notice  had  the  strongest 
overall  impact  on  response  rate  (Dillman,  2000;  Dillman,  Clark  &  Sinclair,  1993). 
Review  of  these  studies  leads  to  the  following  hypothesis  statements: 

Ha3:  Advance  notice  is  related  to  military  member  response  rates,  while 

controlling  for  all  other  variables. 

Ha4:  Follow-up  reminders  are  related  to  response  rates  in  military  samples,  while 
controlling  for  all  other  variables. 

Alternatively,  some  researchers  have  surmised  that  members  of  large 
organizations  (including  the  military)  are  affected  by  the  “good  soldier”  syndrome  in 
which  members  are  expected  to  do  (and  will  do)  what  they  are  told;  these  members  may 
also  marginalize  undesirable  information  in  order  to  present  a  positive  image  for  their 
respective  organizations  (Moradi,  2006;  Organ,  1988).  Additionally,  in  the  previously 
mentioned  study  by  Kaplowitz,  Hadlock  and  Levine  (2004)  where  they  found  that  an 
advance  notice  sent  by  postal  mail  helped  improve  the  response  rate  for  a  web-based 
survey,  their  research  also  found  that  postal  mail  follow-up  reminders  (or  reminder 
notifications)  for  the  same  web-based  survey  were  found  to  be  less  effective.  This 
finding  is  interesting  on  another  level  because  the  purpose  of  their  study  was  to  examine 
the  results  of  a  mixed-mode  type  of  survey,  but  the  fact  that  notification  by  one  mode 
(postal  mail)  giving  notice  about  a  survey  to  be  completed  in  another  mode  (on  the 
Internet)  is  an  interesting  twist  on  what  most  researchers  would  label  “typical”  advance 
notice  or  follow-up  reminders. 


19 


Last,  Cycyota  and  Harrison  (2006)  found  that  advance  notice  and  follow-up 
reminders  did  not  improve  response  rates  for  executives  at  higher  echelons  of  the 
business  world,  according  to  a  meta-analysis  they  perfonned  which  covered  a  ten-year 
time  span  ending  in  early  2000.  These  results  were  almost  identical  to  results  they 
published  in  2002  which  stated  that  traditional  response  rate  factors  which  have  been 
proven  effective  at  the  employee  level  or  consumer  level  (specifically,  advance  notice 
and  follow-up)  did  not  improve  the  survey  response  rate  for  true  executive  level 
populations. 

Given  the  aforementioned  published  research  support,  the  following  hypotheses 
are  given: 

Ho3:  Advance  notice  is  not  related  to  military  member  response  rates,  while 
controlling  for  all  other  variables. 

Ho4:  Follow-up  reminders  are  not  related  to  response  rates  in  military  samples, 
while  controlling  for  all  other  variables. 
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III.  Methodology 


This  chapter  will  describe  the  method  and  analysis  utilized  to  determine  the  effect 
of  the  four  questionnaire  response  rate  factors  on  military-only  samples.  A  meta-analysis 
of  previously  published  research  pertaining  to  military-only  populations  was  perfonned 
to  analyze  the  relative  importance  of  each  of  the  survey  response  rate  factors.  Based  on 
previously  published  meta-analysis  research  (Cycyota  &  Harrison,  2006;  Fox,  Crask  & 
Kim,  1988),  the  following  steps  were  taken.  First,  electronic  sources  of  published  articles 
were  identified  which  contained  pertinent  search  words.  Second,  once  a  potential  article 
was  found,  the  article  was  searched  for  individual  and  combinations  of  search  tenns. 
Third,  any  and  all  of  the  data  contained  in  the  article  which  described  the  independent 
variables  and  the  dependent  variables  were  coded  into  the  data  matrix.  Last,  once  the 
data  matrix  was  complete,  the  data  were  imported  into  SPSS  for  Windows  and  a  linear 
regression  process  was  executed  to  analyze  the  data.  This  process  is  based  on 
recommendations  found  in  research  conducted  by  Viswesvaran  and  Ones  (1995) 
regarding  the  use  of  operational  measures  for  meta-analytical  research.  Cycyota  and 
Harrison  (2006)  continued  this  method  which  allowed  them  to  broadly  interpret 
constructs  and  variables  in  order  to  develop  new  constructs  and  hypotheses  which  lead  to 
conclusions  not  available  in  single  case  analysis;  their  process  focused  on  collecting  and 
analyzing  response  rate  data  instead  of  focusing  on  effect  size  due  to  sample  size.  This 
research  followed  a  similar  process. 
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Sample  Description 

The  target  studies  involved  surveys  of  military-only  populations  that  appeared  in 
a  published  electronic  report  of  some  type  (e.g.,  journal  article,  thesis),  published  after 
1990,  and  made  available  through  an  on-line,  electronic  database.  Questionnaire  formats 
ranged  from  simple  yes/no  response,  fill  in  the  blank,  short  answer  (especially  for 
demographic  data),  to  Likert  scales  for  measuring  perceptions  and  feelings,  up  to  and 
including  free  flow  text  blocks  for  collection  of  lengthy  responses  in  sentence  or 
paragraph  fonn.  The  questionnaires  contained  within  the  published  research  were 
completed  using  pencil  and  paper,  electronically  in  a  format  supported  on  a  computer 
system,  by  utilizing  the  Internet  to  complete  a  web-based  version,  direct  administration 
(in  which  the  researcher  or  designee  handed  out  the  surveys  to  the  target  audience),  or  a 
combination  of  these  modes,  referred  to  as  mixed  mode. 

Only  studies  involving  populations  or  samples  of  members  in  the  following 
military  categories  were  considered  for  this  meta-analysis:  active  duty,  National  Guard, 
Reserve,  cadets  from  the  military  service  academies,  or  veterans  (if  the  survey  pertained 
to  their  time  as  military  members).  Additionally,  several  studies  and  surveys  included 
samples  or  populations  interlaced  with  the  military-only  samples  or  populations,  such  as 
civilians  working  in  military  organizations  or  military  spouses.  These  data  sets  were  only 
used  if  the  military-only  sections  were  broken  out  from  the  non-military  sections  of  the 
survey  population;  in  many  instances  where  the  military  and  non-military  data  were 
combined,  these  two  sectors  were  not  broken  out  during  the  statistical  analysis  in  the 
published  paper,  and  therefore  were  unusable  for  this  particular  research  effort. 
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Multiple  data  sources  were  utilized  during  the  data  search  process.  By  far  the 
largest  source  of  published  research  was  found  in  the  Defense  Technical  Information 
Center  (DTIC)  Scientific  and  Technical  Infonnation  Network  (STINET)  website.  The 
STINET  service  supports  the  Department  of  Defense  community,  but  is  available  to  the 
general  public  and  offers  extensive  search  tools.  Next,  several  electronic  data 
repositories,  such  as  ProQuest  and  ABI/INFORM,  were  utilized.  The  Google  Scholar 
website  (located  at  http://scholar.google.com/)  was  another  source  of  infonnation 
available  to  the  general  public  and  offers  a  search  tool  to  find  full  titles  of  published 
works.  Last,  published  research  stored  on  the  PsycINFO  database  was  searched  for 
potential  data  sources. 

Procedure 

The  procedure  for  locating  published  journal  articles  which  could  possibly  be 
utilized  in  this  research  consisted  of  several  different  search  techniques.  By  utilizing  the 
resources  described  in  the  preceding  paragraphs,  a  very  general  list  of  search  terms  were 
input  into  the  respective  database  search  tools.  Because  this  research  was  to  consider 
response  rate  on  surveys  of  military-only  samples,  keywords  such  as  “military,” 
“sample,”  “response,”  “rate,”  and  “survey”  were  the  first  search  tenns  used  when 
accessing  a  new,  unsearched  database.  As  one  might  imagine,  however,  these  generic 
tenns  usually  returned  many  hundreds  (or  thousands)  of  “hits.”  For  example,  the  DTIC 
database  returns  over  35,000  hits  of  “survey”,  and  almost  8,000  hits  when  “survey”  and 
“military”  are  both  used.  Further  refinement  of  the  search  terms  included  “veterans”, 
“thesis”,  “dissertation”,  “abstract”,  “paper  survey”,  “electronic  survey”,  “web-based 
survey”,  “Internet-based  survey”,  and  “Internet  survey.”  Because  the  term  “survey”  is 
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synonymous  with  the  term  “questionnaire”,  both  terms  were  utilized  in  subsequent 
database  searches.  Also,  to  reduce  the  number  of  hits,  terms  such  as  “telephone”  and 
“interview”  were  considered  undesirable,  and  therefore  the  “not”  option  was  utilized  to 
prevent  return  of  any  possible  hits  which  included  either  or  both  of  these  two  terms. 

Additionally,  on  each  of  the  respective  database  search  pages,  the  “find  full-text 
articles  only”  option  was  selected  whenever  available.  For  this  research,  the  process  for 
reviewing  the  methodology  section  and  attachments  for  each  published  research 
document  required  having  the  full-text  available  for  viewing. 

Measures  and  Coding  Procedures 

Reported  response  rate  was  the  dependent  variable  contained  in  each  of  the 
published  articles;  specifically,  useable  response  rate,  which  is  defined  as  the  total 
number  of  useable  surveys  returned  divided  by  the  total  number  of  surveys  sent.  Also 
collected  was  the  maximum  response  rate,  which  is  the  total  number  of  surveys  returned 
(useable  or  not)  divided  by  the  total  number  of  surveys  sent  (American  Association  for 
Public  Opinion  Research  [AAPOR],  2006). 

Additionally,  the  article  had  to  contain  values  for  at  least  one  of  the  independent 
variables.  Data  were  collected  for  the  response  rate  factors  of  interest:  survey  length, 
survey  delivery  mode  (postal,  e-mail,  web-based,  direct  administered,  or  multi-mode), 
advance  notice,  and  follow-up  reminders.  Survey  length  and  follow-up  reminder 
variables  were  coded  as  interval  data.  Survey  length  is  merely  the  number  of  questions 
contained  in  the  survey.  Follow-up  reminders  were  coded  either  zero  (for  missing  or  not 
utilized),  or  one  through  four  to  indicate  the  number  of  follow-up  reminders  sent  as  part 
of  the  respective  study.  The  other  independent  variables  (survey  delivery  mode  and 
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advance  notice)  were  coded  as  dummy  variables:  zero  for  not  present  or  not  utilized,  or 
one  for  present  or  utilized. 


Related  data  were  also  collected  and  recorded  on  variables  for  branch  of  military 
service  (e.g.,  USAF,  USMC),  sponsorship  level  (subjectively  assigned  a  value  of  high, 
medium,  or  low),  sponsor  details  (office  symbol  of  sponsor,  if  available),  sampling 
technique  (e.g.,  random,  stratified  random,  convenience),  and  other/notes.  Sponsorship 
level,  although  subjectively  assigned,  utilized  the  following  coding  scheme:  “high”  was 
reserved  for  the  President  of  the  United  States,  the  service  secretaries,  and  large,  national 
organizations;  “medium”  was  assigned  to  sponsors  at  the  major  command  (MAJCOM) 
level,  or  commanders  in  wings,  groups,  or  squadrons;  “low”  was  given  when  either  no 
sponsor  was  assigned  or  the  sponsor  was  another  student  or  only  the  student  was 
identified  (i.e.,  no  sponsor). 
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IV.  Data  Analysis  and  Results 


The  data  analysis  for  the  collected  questionnaire  data  began  with  an  examination 
of  the  entire  data  matrix  to  determine  which  specific  variables  would  be  analyzed.  As 
previously  described,  some  data  were  collected  in  the  data  matrix  in  order  to  enhance  the 
overall  data  richness  and  would  not  be  considered  for  the  final  analysis.  However,  this 
non-analyzed  data  might  offer  additional  information  for  consideration  during  the  final 
chapter  of  this  thesis. 

The  final  independent  variables  selected  for  analysis  were  questionnaire  length 
(number  of  questions),  method  of  delivery  (postal,  e-mail,  web-based,  direct 
administration,  or  multi-mode),  use  of  advance  notice  notifications,  and  use  of  follow-up 
reminders.  These  variables  were  selected  and  copied  from  the  main  worksheet  in  the  data 
matrix  file,  and  pasted  into  a  new  Excel  worksheet.  Portions  of  the  data  in  this  worksheet 
were  then  recoded  for  entry  into  SPSS  14  for  Windows,  changing  all  variations  of 
dichotomous  variables  (e.g.,  Y/N)  into  “0”  and  “1”.  No  recoding  was  required  for  the 
dependent  variable  (response  rate)  and  one  of  the  independent  variables  (survey  length). 

Data  Analysis 

For  the  first  part  of  this  section,  the  analysis  only  compared  the  effect  of  one  of 
the  four  response  rate  factors,  delivery  mode,  on  response  rate.  The  second  part  of  this 
section  compared  the  effect  of  all  of  the  response  rate  factors  on  response  rate  by  utilizing 
regression  analysis  tools  in  SPSS. 

For  the  deliver  mode  analysis,  73  records  contained  useable  response  rate  data  (n 
=  73).  This  analysis  required  creation  of  a  fifth  variable  named  “multi  mode”  to  account 
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for  a  double  count  of  data  if  more  than  one  delivery  mode  was  used  in  the  case  (i.e.,  both 
postal  and  web-based.)  When  this  situation  occurred  in  the  data,  the  “multi  mode” 
variable  was  set  to  “1”  and  each  of  the  delivery  mode  variables  which  previously  equaled 
“1”  were  now  set  to  zero.  The  table  below  shows  the  response  rate  for  each  respective 
delivery  mode,  and  an  overall  response  rate  of  69%  for  these  surveys.  Upon  initial 


review,  e-mail  and  direct  administration  surveys  appeared  to  have  the  highest  response 


rate  of  88%.  However,  these  results  are  not  weighted  for  individual  sample  size. 


Table  1.  Response  Rates  by  Delivery  Mode 


Response  Rates  by  Delivery  Mode 

n 

RR  avg 

Postal 

25 

63.0% 

E-Mail 

2 

88.9% 

Web 

22 

44.8% 

DA 

13 

88.3% 

Multi_Mode 

11 

63.5% 

Overall  Average  RR: 

69.7% 

Total  Questionnaires: 

73 

Further  analysis  of  more  complex  combinations  of  delivery  mode  and  either 
advance  notice  or  follow-up  reminders  are  displayed  below. 

_ Table  2.  Response  Rates  by  Delivery  Mode  and  Advance  Notice _ 

j  Response  Rates  by  Delivery  Mode  and  Advance  Notice 


Postal  E-Mail  Web  DA  Multi  Mode 

n  =  73  25  2  22  13  11 

RR  avg  63.0% _ 88.9% _ 44.8% _ 88.3% _ 63.5% 


Advance  Notice 

40 

64.4% 

64.6% 

12 

100.0% 

1 

53.8% 

14 

86.4% 

4 

66.8% 

9 

No  Advance  Notice 

33 

60.9% 

61.6% 

13 

77.8% 

1 

29.1% 

8 

89.2% 

9 

48.7% 

2 

Table  3.  Response  Rates  by  Delivery  Mode  and  Follow-up  Reminders 

Response  Rates  by  Delivery  Mode  and  Follow-up  Reminders 


Postal  E-Mail  Web  DA  Multi  Mode 

n  =  73  25  2  22  13  11 

RR  avg  63.0% _ 88.9% _ 44.8% _ 88.3% _ 63.5% 


Follow-up  Reminders 

31 

58.0% 

70.5% 

13 

0 

32.8% 

9 

0 

65.3% 

9 

No  Follow-ups 

42 

66.3% 

55.0% 

12 

88.9% 

2 

53.1% 

13 

88.3% 

13 

55.2% 

2 
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Additionally,  this  spreadsheet  was  utilized  to  create  an  F  distribution  for  an 


analysis  of  variance  comparison  of  the  independent  variables  which  composed  the 
delivery  mode.  The  results  from  this  analysis  are  shown  below. 


Table  4.  Analysis  of  Variance 


Source 

Sum  of 
Squares 

df 

s2 

F  test 

F  crit 

a  =  .10 

Between 

4.0000 

4 

0.5101 

7.7293 

2.03 

Within 

0.1294 

68 

0.0660 

Total 

4.1294 

72 

Based  on  the  analysis,  the  results  indicate  Ftest  is  greater  than  Fcrit  (Ftest  >  Fcrit)  for 
Fcnt  (.10, 4, 68),  which  suggests  the  means  of  the  separate  groups  are  not  similar  (Agresti  & 
Finlay,  1997). 

Next,  all  of  the  response  rate  factors  were  analyzed  for  effect  on  response  rate. 
SPSS  was  utilized  to  import  the  recoded  data  for  the  dependent  and  all  the  independent 
variables  from  the  Excel  spreadsheet.  The  data  were  then  analyzed,  and  the  results  are 
presented  in  the  remainder  of  this  chapter. 


A  descriptive  analysis  was  run  in  SPSS  in  order  to  perform  a  quick  quality  check 


of  the  data  before  further  analysis  was  preformed.  N,  range,  minimum  and  maximum 


were  reviewed  and  found  to  be  acceptable. 

_ Table  5.  Descriptive  Statistics 


Descriptive  Statistics 


N 

Range 

Minimum 

Maximum 

Sum 

Mean 

Std. 

Variance 

Statistic 

Statistic 

Statistic 

Statistic 

Statistic 

Statistic 

Std.  Error 

Statistic 

Statistic 

RR_Max 

73 

1.0 

.0 

1.0 

45.9 

.628 

.0343 

.2930 

.086 

Length 

73 

586 

10 

596 

5814 

79.64 

1 1 .556 

98.737 

9749.010 

Postal 

73 

1 

0 

1 

30 

.41 

.058 

.495 

.245 

E_Mail 

73 

1 

0 

1 

5 

.07 

.030 

.254 

.065 

Web 

73 

1 

0 

1 

30 

.41 

.058 

.495 

.245 

Direct_Admin 

73 

1 

0 

1 

22 

.30 

.054 

.462 

.213 

Multi_Mode 

73 

1 

0 

1 

11 

.15 

.042 

.360 

.130 

Adv_Notice 

73 

1 

0 

1 

40 

.55 

.059 

.501 

.251 

Followup 

73 

4 

0 

4 

48 

.66 

.107 

.916 

.839 

Valid  N  (listwise) 

73 
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Based  on  data  contained  in  the  correlation  table,  the  following  correlations 
between  either  web-based  delivery  mode  and  response  rate,  or  direct  administration 
delivery  mode  and  response  rate,  were  found  to  be  significant: 

1)  Web-based  delivery  mode  (IV)  and  response  rate  (DV).  With  a  correlation 
coefficient  of  r  =  -.32  (p  <  .01),  the  data  indicate  a  significant,  medium  strength  negative 
relationship  between  web-based  delivery  mode  and  response  rate. 

2)  Direct  administration  delivery  mode  (IV)  and  response  rate  (DV).  With  a 
correlation  coefficient  of  r  =  .40  (p  <  .01),  the  data  indicate  a  significant,  fairly  strong 
positive  relationship  between  direct  administration  delivery  mode  and  response  rate. 


Table  6.  Correlations 


Correlations 

_ 

Mean 

Std  Dev 

1 

2 

3 

4 

5 

6 

7 

8 

0.2930 

1 

79.64 

98.737 

.192 

1 

0.41 

0.495 

-.098 

-.147 

1 

0.07 

0.254 

.035 

-.067 

-.116 

1 

5 

0.41 

0.495 

-.321** 

.211 

-.584** 

-.006 

1 

6 

0.30 

0.462 

.404** 

-.046 

-.367** 

.176 

-.185 

1 

sa 

0.55 

0.501 

.059 

.206 

-.080 

.028 

.255* 

-.003 

1 

H 

0.66 

0.916 

-.153 

.019 

.100 

.042 

.345** 

-.048 

.475** 

1 

'*  Correlation  is  significant  at  the  0.01  level  (2-tailed). 
*  Correlation  is  significant  at  the  0.05  level  (2-tailed). 


1  RR_Max 

2  Length 

3  Postal 

4  E_Mail 

5  Web 

6  Direct_Admin 

7  Adv_Notice 

8  Followup 


Continuing  the  regression  analysis  from  output  generated  in  SPSS,  the  following 


tables  were  generated  and  are  analyzed  in  the  following  paragraphs. 


2 

First,  the  model  summary  is  shown.  R  ,  the  multiple  coefficient  of  determination, 


was  .34;  this  indicates  that  the  independent  variables  in  the  model  explain  34%  of  the 


variance  in  the  dependent  variable,  response  rate. 
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Table  7.  Model  Summary 


Model  Summar^ 


Model 

R 

R  Square 

Adjusted 

R  Square 

Std.  Error  of 
the  Estimate 

Change  Statistics 

Durbin- 

Watson 

R  Square 
Change 

F  Change 

dfl 

df2 

Sig.  F  Change 

1 

.583a 

.340 

.269 

.2505 

.340 

4.784 

7 

65 

.000 

1.758 

a-  Predictors:  (Constant),  Followup,  Length,  Direct_Admin,  E_Mail,  Postal,  Adv_Notice,  Web 
b.  Dependent  Variable:  RR_Max 


Second,  the  ANOVA  table  shown  below  confirms  34%  of  the  variance  is 


accounted  for  in  this  model  (2.102  /  6.182  =  .340).  Additionally,  the  F  value  indicates  the 


ability  of  the  independent  variables  to  predict  the  dependent  variable  in  the  model  is 


significant  and  the  variation  is  not  due  to  chance  (F  =  4.784;  p  <  .001,  n  =  72). 

Table  8.  ANOVA 


ANOVAP 


Model 

Sum  of 
Squares 

df 

Mean  Square 

F 

Sig. 

1  Regression 

2.102 

7 

4.784 

,000a 

Residual 

4.080 

65 

Total 

6.182 

72 

a-  Predictors:  (Constant),  Followup,  Length,  Direct_Admin,  E_Mail,  Postal,  Adv_Notice, 
Web 

b-  Dependent  Variable:  RR_Max 


Next,  a  coefficients  table  was  compiled  in  order  to  find  the  coefficients  for  each 
of  the  variables  and  their  respective  significance  levels.  The  table  below  shows  that 
although  seven  independent  variables  were  entered  in  the  model,  only  two  were  found  to 
be  significant:  survey  length  and  web-based  delivery  mode  (p  <  .05).  However,  while 
survey  length  was  found  to  be  significant,  it  has  such  a  low  value  that  it  will  have 
minimal  impact  on  the  overall  output  of  the  model  and,  therefore,  is  of  minimal  practical 
significance. 
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Table  9.  Coefficients 


Coefficients3 


Model 

Unstandardized 

Coefficients 

Standardized 

Coefficients 

t 

Sig. 

B 

Std.  Error 

Beta 

1  (Constant) 

.695 

.101 

6.876 

.000 

Length 

.001 

.000 

.246 

2.330 

.023 

Postal 

-.176 

.107 

-.297 

-1.645 

.105 

E_Mail 

-.032 

.119 

-.028 

-.271 

.787 

Web 

-.321 

.107 

-.543 

-2.994 

.004 

Direct_Admin 

.135 

.086 

.212 

1.557 

.124 

Adv_Notice 

.068 

.069 

.117 

.988 

.327 

Followup 

.005 

.044 

.015 

.112 

.911 

a.  Dependent  Variable:  RR_Max 


Last,  the  residuals  from  the  regression  process  were  analyzed  to  ensure  their 


distribution  was  normal  or  nearly  normal,  which  is  a  requirement  for  the  ANOVA 


analysis.  The  analysis  confirmed  the  residual  distributions  met  this  requirement. 


Results  and  Hypothesis  Analysis 

Hypothesis  1  tested  the  relationship  between  survey  length  and  response  rate  in 
military  samples,  while  controlling  for  all  other  variables.  Although  a  significant 
bivariate  correlation  between  survey  length  and  response  rate  was  not  found,  results  from 
further  regression  analysis  indicate  that  survey  length  was  significantly  related  to 
response  rate.  However,  with  a  value  of  only  .001,  even  though  significant  (p  <  .05),  the 
overall  impact  of  survey  length  on  response  rate  is  negligible  (i.e.,  minimal  practical 
significance.)  Due  to  the  significance  finding,  though,  the  null  hypothesis  H()|,  which 

states  survey  length  is  not  related  to  response  rates  in  military  samples  while  controlling 
for  all  other  variables,  must  therefore  be  rejected.  The  alternate  hypothesis  Ha,,  which 
states  survey  length  is  related  to  response  rates  in  military  samples  while  controlling  for 
all  other  variables,  is  supported. 
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Hypothesis  2  tested  if  delivery  mode  influences  response  rates  in  military 
samples,  while  controlling  for  all  other  variables.  First,  the  results  from  the  F  distribution 
found  that  the  means  between  the  different  survey  delivery  modes  were  probably  not 
equal.  Next,  only  web-based  delivery  mode  was  found  to  have  a  significant  correlation, 
and  it  was  found  to  be  related  to  response  rate  (r  =  -.32,  p  <  .01).  These  results  indicate  a 
significant,  negative  correlation.  Last,  further  regression  analysis  presented  two 
independent  variables  found  to  be  statistically  significant,  one  of  which  was  a  delivery 
mode  variable.  Web-based  delivery  mode  was  found  to  have  the  largest  regression 
coefficient  of  -.32  (p  <  .01).  Based  on  these  data,  delivery  mode  appears  to  have  an 
impact  on  response  rate.  Therefore,  reject  the  null  hypothesis  H0n  which  supposed  that 
delivery  mode  does  not  influence  response  rates  in  military  samples,  while  controlling  for 
all  other  variables.  Since  the  null  was  rejected,  HAn  must  be  accepted;  HAl  stated  that 

delivery  mode  does  influence  response  rates  in  military  samples,  while  controlling  for  all 
other  variables. 

Hypothesis  3  tested  if  advance  notice  is  related  to  military  member  response  rates, 
while  controlling  for  all  other  variables.  Based  on  the  analysis  from  the  regression 
procedures,  no  statistically  significant  results  were  found  to  reject  the  null  hypothesis 
Hor  Therefore,  fail  to  reject  Ho3,  which  stated  advance  notice  is  not  related  to  military 
member  response  rates,  while  controlling  for  all  other  variables.  Also,  fail  to  support 
Ha3,  which  stated  advance  notice  is  related  to  military  member  response  rates,  while 
controlling  for  all  other  variables. 
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Hypothesis  4  tested  if  follow-up  reminders  are  related  to  military  member 
response  rates,  while  controlling  for  all  other  variables.  Based  on  analysis  of  the 
regression  procedures,  no  statistically  significant  results  were  found  to  reject  the  null 
hypothesis  Ho4.  Therefore,  fail  to  reject  Ho4,  which  stated  follow-up  reminders  are  not 
related  to  military  member  response  rates,  while  controlling  for  all  other  variables.  Also, 
fail  to  support  Ha4,  which  stated  follow-up  reminders  are  related  to  military  member 

response  rates,  while  controlling  for  all  other  variables. 
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V.  Discussion 


The  overall  goal  of  this  research  was  to  evaluate  if  response  rates  from  military- 
only  samples  react  to  factors  found  to  impact  response  rates  in  general  populations.  The 
four  factors  selected  for  consideration  in  this  study  included  survey  length,  delivery 
mode,  advance  notice,  and  follow-up  reminders.  The  only  supportable  findings  for  this 
sample  of  cases  were  that  survey  length  and  survey  delivery  mode  may  influence 
response  rate  in  military-only  samples.  Specifically,  this  meta-analysis  suggests  that 
military  members  respond  more  to  the  length  of  a  survey  and  the  method  of  survey 
delivery  rather  than  the  use  of  advance  notices  and  follow-up  reminders. 

The  finding  for  hypothesis  1  regarding  the  impact  of  survey  length  on  response 
rate  seems  supported  by  previous  research  by  Dillman,  Sinclair,  and  Clark  (1993),  Bogen 
(1996),  Sheehan  (2001),  and  Smith,  Olah,  Hansen,  and  Cumbo  (2003).  Although  the 
actual  impact  found  in  this  research  was  found  to  be  statistically  significant,  yet  not 
practically  significant,  these  previous  researchers  arrived  at  stronger  findings  of  the 
relationship  between  survey  length  and  response  rate. 

The  finding  of  hypothesis  2  pertaining  to  how  survey  delivery  mode  influences 
response  rate  in  military  samples  is  a  little  more  difficult  to  quantify.  As  previously 
stated,  the  web-based  survey  delivery  mode  was  found  to  have  a  negative  correlation  with 
response  rate,  and  a  negative  regression  coefficient  for  this  mode  was  found  during 
further  regression  analysis.  These  results  may  indicate  that  while  the  web-based  survey  is 
typically  easier  to  generate  and  a  web  link  can  easily  be  sent  to  a  very  large  audience, 
perhaps  this  is  an  example  of  casting  the  net  too  wide  and  too  frequently,  and  a  lower 
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response  rate  for  web-based  surveys  is  the  result.  MacElroy,  Milucki,  and  McDowell 
(2002)  found  an  increase  in  the  use  of  web-based  surveys,  which  may  lead  to  saturation. 
Regarding  different  delivery  modes,  several  sources  cited  differences  in  responses  when 
comparing  one  delivery  mode  against  another  (McCoy,  Marks,  Carr  &  Mbarika,  2004; 
Webster  &  Compeau,  1996). 

Hypothesis  3  and  4,  pertaining  to  use  of  advance  notice  and  follow-up  reminders, 
were  found  to  apparently  not  impact  response  rate  in  this  research.  Perhaps  these  results 
are  related  to  similar  findings  by  other  researchers.  For  example,  since  military-only 
samples  were  considered,  some  suggest  that  military  members  are  expected  to  be  “good 
soldiers”,  carry  out  their  duties  without  being  reminded,  and  present  a  positive,  “leaning 
forward,”  military  image;  follow-up  reminders  might  have  little,  if  any,  impact  if  military 
members  immediately  complete  surveys  upon  initial  notification  (Moradi,  2006;  Organ, 
1988).  Cycyota  and  Harrison  (2006)  found  similar  results  from  a  study  among  business 
executives:  advance  notice  and  follow-up  reminders  were  inefficient.  Last,  in  2004, 
Kaplowitz,  Hadlock,  and  Levine  detennined  follow-up  reminders  were  ineffective  in  a 
review  of  multi-mode  surveys.  Each  of  these  researchers  found  little  utility  for  advance 
notice  or  follow-up  reminders  in  these  situations. 

Recommendations 

Based  on  the  findings  in  this  research,  several  areas  are  recommended  for 
researchers  considering  use  of  a  survey  or  questionnaire  as  part  of  a  data  collection 
project.  First,  since  survey  length  was  found  to  be  a  factor  in  this  research,  potential  data 
collectors  contemplating  use  of  a  survey  for  data  collection  must  consider  the  effect  of 
survey  length.  Another  consideration  closely  related  to  survey  length  (i.e.,  number  of 
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questions)  is  the  anticipated  time  factor  for  users  to  complete  a  survey.  Researchers 
should  be  cognizant  of  these  areas.  Second,  survey  delivery  mode  was  also  found  to  be 
significant.  Potential  researchers  should  carefully  consider  the  mode  (or  modes)  being 
considered  for  potential  survey  delivery.  Is  a  strictly  web-based  delivery  mode  being 
considered  since  it  is  easier  and  cheaper?  Is  the  web  host  for  the  web-based  survey 
competent  in  building  a  web-based  survey  and  collecting  the  results?  Researchers  should 
carefully  consider  all  survey  delivery  modes  available,  and  then  choose  the  best  mode  (or 
modes)  for  their  particular  subject  and  target  audience.  Last,  one  related  area  not 
specifically  addressed  in  the  four  hypothesis  statements:  survey  design.  Researchers  are 
encouraged  to  follow  the  guidance  given  by  some  of  the  industry  leaders  cited  in  this 
research.  This  will  help  the  survey  to  be  easier  for  the  target  audience  to  use  and  will 
assist  in  the  process  of  collecting  the  desired  data  from  the  target  audience.  A  well- 
written,  professional-looking  survey  tells  the  audience  the  topic  is  relevant,  substantial, 
and  worth  their  time  to  participate.  Each  of  these  recommendations,  based  on  the 
research  conducted  for  the  literature  review,  should  lead  to  a  positive  impact  on  survey 
response  rate. 

Limitations 

This  research  contained  several  limitations.  First,  when  conducting  the  search  for 
previously  published  cases  which  utilized  a  survey  as  part  of  the  data  collection  effort, 
only  electronically  available  files  offered  in  a  “full  text”  output  were  included  as  part  of 
the  search  process.  This  almost  certainly  reduced  the  number  of  available  of  cases  to  be 
considered.  Second,  this  research  made  no  attempt  to  analyze  or  quantify  errors  which 
may  have  existed  within  each  of  the  selected  cases,  such  as  sampling  error  or  non- 
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response  error,  or  social  desirability  considerations,  such  as  acquiescence;  the  impact  of 
the  human  factor  associated  with  each  of  the  surveys  utilized  in  each  of  the  selected  cases 
is  beyond  the  scope  of  this  research.  Third,  a  larger  sample  size  would  have  been 
desirable;  perhaps  some  of  the  non-significant  findings  may  have  become  significant  with 
more  cases  to  analyze.  Last,  a  general  lack  of  consistency  among  the  reviewed  and 
selected  cases  pertaining  to  a  survey  methodology  was  evident;  frequently,  infonnation 
about  the  specific  survey  methodology  utilized,  survey  design,  and  survey  results  were 
found  scattered  throughout  the  individual  cases  instead  of  being  explicitly  detailed  and 
centralized  in  what  is  typically  the  methodology  and  results  sections.  A  standardized 
process  for  survey  methodology,  survey  design,  and  survey  data  reporting  would  greatly 
improve  this  process  if  additional  research  were  performed  in  this  area. 

Suggestions  for  Future  Research 

The  lack  of  adequate  research  into  this  area  becomes  apparent  when  seeking 
information  on  how  military-only  samples  react  to  traditional  survey  response  rate 
factors.  With  a  potentially  dwindling  base  of  military  personnel,  and  an  overall  decline  in 
survey  response  rates  in  the  general  population,  it  becomes  even  more  important  to  gain 
further  understanding  of  how  to  increase  response  rates  from  military  members.  Studies 
conducted  on  the  general  public  have  found  many  factors  which  influence  survey 
response,  but  additional  research  is  required  to  understand  how  these  factors,  if  any,  can 
be  applied  to  a  military-only  sample. 
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Conclusion 


The  primary  goal  of  this  research  was  to  attempt  to  verify  if  general  population 
survey  response  techniques  work  for  military  surveys.  The  preceding  paragraphs  indicate 
significant  results  were  found,  but  more  research  is  required  in  this  area.  A  secondary 
goal  was  to  provide  a  quick  “one-stop  shopping”  resource  for  two  groups  within  our 
military:  1)  military  members  currently  in  the  field  who  may  be  tasked  by  higher 
authorities  to  generate  a  survey,  analyze  the  results,  and  present  the  findings;  and  2) 
military  members  who  are  advancing  their  educational  goals  and  need  to  generate  a 
survey  as  part  of  either  an  off-duty  education  program  or  a  dedicated  educational 
program  such  as  an  advanced  degree  program  or  a  professional  military  education 
program.  Hopefully,  the  preceding  sections  will  provide  an  entry  point  into  a  general 
overview  of  military  survey  response  rate  considerations,  as  well  as  identify  in-depth 
resources  written  by  industry  experts.  In  the  end,  finding  ways  to  increase  survey 
response  rates  among  military-only  populations  will  give  military  and  civilian  leaders  the 
information  required  to  continually  improve  our  smaller,  leaner  military  and  ensure  our 
position  as  the  greatest  military  in  the  world. 
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